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From Biological 
to Artificial Neuron Model 


Martin Gardner in his book titled 'The Annotated Snark" has the following note for the 
last illustration sketched by Holiday for Lewis Carroll's nonsense poem 'The Hunting of 
the Snark': 

"64. Holiday's illustration for this scene, showing the Bellman ringing a knell for the 
passing of the Baker, is quite a remarkable puzzle picture. Thousands of readers must 
have glanced at this picture without noticing (though they may have shivered with 
subliminal perception) the huge, almost transparent head of the Baker, abject terror on his 
features, as a gigantic beak (or is it a claw?) seizes his wrist and drags him into the 
ultimate darkness." 

You also may have not noticed that the face at the first glance, however with a little more 
care, you will it easily. As a further note on Martin Gardner's note, we can ask the 
question "Is there is any conventional computer at present with the capability of 
perceiving both the trees and Baker's transparent head in this picture at the same time?" 
Most probably, the answer is no. 

Although such a visual perception is an easy task for human being, we are faced with 
difficulties when sequential computers are to be programmed to perform visual 
operations. 

In a conventional computer, usually there exist a single processor implementing a 
sequence of arithmetic and logical operations, nowadays at speeds approaching billion 
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operations per second. However this type of devices have ability neither to adapt their 
structure nor to learn in the way that human being does. 

There is a large number of tasks for which it is proved to be virtually impossible to 
device an algorithm or sequence of arithmetic and/or logical operations. For example, in 
spite of many attempts, a machine has not yet been produced which will automatically 
read handwritten characters, or recognize words spoken by any speaker let alone translate 
from one language to another, or drive a car, or walk and run as an animal or human 
being [Hecht-Nielsen 88] 

What makes such a difference seems to be neither because of the processing speed of the 
computers nor because of their processing ability. Today’s processors have a speed 10 (1 
times faster than the basic processing elements of the brain called neuron. When the 
abilities are compared, the neurons are much simpler. The difference is mainly due to the 
structural and operational trend. While in a conventional computer the instructions are 
executed sequentially in a complicated and fast processor, the brain is a massively 
parallel interconnection of relatively simple and slow processing elements. 

1.1. Biological Neuron 

It is claimed that the human central nervous system is comprised of about 1,3x10 10 
neurons and that about 1x10 10 of them takes place in the brain. At any time, some of 
these neurons are firing and the power dissipation due this electrical activity is estimated 
to be in the order of 10 watts. Monitoring the activity in the brain has shown that, even 
when asleep, 5x1 tf nerve impulses per second are being relayed back and forth between 
the brain and other parts of the body. This rate is increased significantly when awake 
[Fischer 1987]. 

A neuron has a roughly spherical cell body called soma (Figure 1.1). The signals 
generated in soma are transmitted to other neurons through an extension on the cell body 
called axon or nerve fibres. Another kind of extensions around the cell body like bushy 
tree is the dendrites, which are responsible from receiving the incoming signals 
generated by other neurons. [Noakes 92] 
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An axon (Figure 1.2), having a length varying from a fraction of a millimeter to a meter 
in human body, prolongs from the cell body at the point called axon hillock. At the other 
end, the axon is separated into several branches, at the very end of which the axon 
enlarges and forms tenninal buttons. Tenninal buttons are placed in special structures 
called the synapses which are the junctions transmitting signals from one neuron to 
another (Figure 1.3). A neuron typically drive 1 (P to I ()4 synap tic junctions 



Figure 1.2. Axon 

The synaptic vesicles holding several thousands of molecules of Chemical transmitters, 
take place in tenninal buttons. When a nerve impulse arrives at the synapse, some of 
these Chemical transmitters are discharged into synaptic cleft, which is the narrow gap 
between the terminal button of the neuron transmitting the signal and the membrane of 
the neuron receiving it. In general the synapses take place between an axon branch of a 
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neuron and the dendrite of another one. Although it is not very cornmon, synapses may 
also take place between two axons or two dendrites of different cells or between an axon 
and a cell body. 



Neurons are covered with a semi-permeable membrane, with only 5 nanometer 
thickness. The membrane is able to selectively absorb and reject ions in the intracellular 
fluid. The membrane basically acts as an ion pump to maintain a different ion 
concentration between the intracellular fluid and extracellular fluid. While the sodium 
ions are continually removed from the intracellular fluid to extracellular fluid, the 
potassium ions are absorbed from the extracellular fluid in order to maintain an 
equilibrium condition. Due to the difference in the ion concentrations inside and outside, 
the cell membrane become polarized. In equilibrium the interior of the cell is observed 
to be 70 milivolts negative with respect to the outside of the cell. The mentioned 
potential is called the resting potential. 

A neuron receives inputs from a large number of neurons via its synaptic connections. 
Nerve signals arriving at the presynaptic cell membrane cause Chemical transmitters to 
be released in to the synaptic cleft. These Chemical transmitters diffuse across the gap 
and join to the postsynaptic membrane of the receptor site. The membrane of the post- 
synaptic cell gathers the Chemical transmitters. This causes either a decrease or an 
increase in the soma potatial, called graded potantial, depending on the type of the 
Chemicals released in to the synaptic cleft. The kind of synapses encouraging 
depolarization is called excitatory and the others discouraging it are called inhibitory 
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synapses. If the decrease in the polarization is adequate to exceed a threshold then the 
post-synaptic neuron fires. 

The arrival of impulses to excitatory synapses adds to the depolarization of soma while 
inhibitory effect tends to cancel out the depolarizing effect of excitatory impulse. In 
general, although the depolarization due to a single synapse is not enough to fire the 
neuron, if some other areas of the membrane are depolarized at the same time by the 
arrival of nerve impulses through other synapses, it may be adequate to exceed the 
threshold and fire. 

At the axon hillock, the excitatory effects result in the interruption the regular ion 
transportation through the cell membrane, so that the ionic concentrations immediately 
begin to equalize as ions diffuse through the membrane. If the depolarization is large 
enough, the membrane potential eventually collapses, and for a short period of time the 
intemal potential becomes positive. The action potential is the name of this brief 
reversal in the potential, which results in an electric current flowing from the region at 
action potential to an adj acent region on axon with a resting potential. This current 
causes the potential of the next resting region to change, so the effect propagates in this 
manner along the axon membrane. 
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Once an action potential has passed a given point, it is incapable of being reexcited for a 
while called refractory period. Because the depolarized parts of the neuron are in a State 
of recovery and can not immediately become active again, the pulse of electrical activity 
always propagates in only forward direction. The previously triggered region on the 
axon then rapidly recovers to the polarized resting State due to the action of the sodium 
potassium pumps. The refractory period is about 1 milliseconds, and this limits the nerve 
pulse transmission so that a neuron can typically fire and generate nerve pulses at a rate 
up to 1000 pulses per second. The number of impulses and the speed at which they 
arrive at the synap tic junctions to a particular neuron determine whether the total 
excitatory depolarization is sufficient to cause the neuron to fire and so to send a nerve 
impulse down its axon. The depolarization effect can propagate along the soma 
membrane but these effects can be dissipated before reaching the axon hillock. 
However, once the nerve impulse reaches the axon hillock it will propagate until it 
reaches the synapses where the depolarization effect will cause the release of Chemical 
transmitters into the synaptic cleft. 

The axons are generally enclosed by myelin sheath that is made of many layers of 
Schwann cells promoting the growth of the axon. The speed of propagation down the 
axon depends on the thickness of the myelin sheath that provides for the insulation of 
the axon from the extracellular fluid and prevents the transmission of ions across the 
membrane. The myelin sheath is interrupted at regular intervals by narrow gaps called 
nodes of Ranvier where extracellular fluid makes contact with membrane and the 
transfer of ions occur. Since the axons themselves are poor conductors, the action 
potential is transmitted as depolarizations occur at the nodes of Ranvier. This happens in 
a sequential manner so that the depolarization of a node triggers the depolarization of 
the next one. The nerve impulse effectively jumps from a node to the next one along the 
axon each node acting rather like a regeneration amplifier to compensate for losses. 
Once an action potential is created at the axon hillock, it is transmitted through the axon 
to other neurons. 

It is mostly tempted to conclude the signal transmission in the nervous system as having 
a digital nature in which a neuron is assumed to be either fully active or inactive. 
However this conclusion is not that correct, because the intensity of a neuron signal is 
coded in the frequency of pulses. A better conclusion would be to interpret the 
biological neural systems as if using a form of pulse frequency modulation to transmit 
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information. The nerve pulses passing along the axon of a particular neuron are of 
approximately constant amplitude but the number generated pulses and their time 
spacing is controlled by the statistics associated with the arrival at the neuron's many 
synaptic junctions of sufficient excitatory inputs [Muller and Reinhardt 90]. 

The representation of biophysical neuron output behavior is shown schematically in 
Figure 1.5 [Kandel 85, Sejnowski 81]. At time t=0 a neuron is excited; at time T, 
typically it may be of the order of 50 milliseconds, the neuron lires a train of impulses 
along its axon. Each of these impulses is practically of identical amplitude. Some time 
later, say around t=T+x, the neuron may lire another train of impulses, as a result of the 
same excitation, though the second train of impulses will usually contain a smaller 
number. Even when the neuron is not excited, it may send out impulses at random, 
though much less frequently than the case when it is excited. 
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Figure 1.5. Representation of biophysical neuron output signal after excitation at tine t= 0 

A considerable amount of research has been perfonned aiming to explain the 
electrochemical structure and operation of a neuron, however still remains several 
questions, which need to be answered in future. 

1.2. Artificial Neuron Model 

As it is mentioned in the previous section, the transmission of a signal from one neuron to 
another through synapses is a complex Chemical process in which specilic transmitter 
substances are released from the sending side of the junction. The effect is to raise or 
lower the electrical potential inside the body of the receiving cell. If this graded potential 
reaches a threshold, the neuron lires. It is this characteristic that the artificial neuron 
model proposed by McCulloch and Pitts, [McCulloch and Pitts 1943] attempt to 
reproduce. The neuron model shown in Figure 1.6 is the one that widely used in artificial 
neural networks with some minor modifications on it. 
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Figure 1.6. Artificial Neuron 

The artificial neuron given in this figure has N input, denoted as uj, uy, ...u N . Each line 
connecting these inputs to the neuron is assigned a weight, which are denoted as wj, W 2 , 
wn respectively. Weights in the artificial model correspond to the synaptic connections 
in biological neurons. The threshold in artificial neuron is usually represented by 9 and 
the activation corresponding to the graded potential is given by the formula: 

N 

d = ('y' i W;Uj)+ 9 (1.2.1) 

.7 = 1 

The inputs and the weights are real values. A negative value for a weight indicates an 
inhibitory connection while a positive value indicates an excitatory one. Although in 
biological neurons, 7? has a negative value, it may be assigned a positive value in artificial 
neuron models. If 9 is positive, it is usually referred as bias. For its mathematical 
convenience we will use (+) sign in the activation formula. Sometimes, the threshold is 
combined for simplicity into the summation part by assuming an imaginary input uo =+l 
and a connection weight wo = 9. Hence the activation formula becomes: 

N 

a - ' S F. w i u i (1-2-2) 

.7=0 

The output value of the neuron is a function of its activation in an analogy to the firing 
frequency of the biological neurons: 
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X = f(a) (1.2.3) 

Furthermore the vector notation 

o = w T u + # (1-2.4) 

is useful for expressing the activation for a neuron. Here, the element of the input 
vector u is uj and the j^ 1 element of the weight vector of w is wj. Both of these vectors 
are of size N. Notice that, w^u is the inner product of the vectors w and u, resulting in a 
scalar value. The inner product is an operation defined on equal sized vectors. In the case 
these vectors have unit length, the inner product is a measure of similarity of these 
vectors. 

Originally the neuron output function j(a) in McCulloch Pitts model proposed as 
threshold function, however linear, ramp and sigmoid and functions (Figure 1.6.) are also 
widely used output functions: 

Linear: 


/ (o) = m 


(1.2.5) 


Threshold: 


/(«) 


/O a < 0 
\l 0 < a 


Ramp: 


lO a< 0 

f(a) = ia//c 0 <a< k 

\l k < a 


(1.2.6) 


(1.2.7) 
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Figure 1.7. Some neuron output fiinctions 


Sigmoid: 

/(«) = (1-2.8) 

l + e~ m 

Though its simple structure, McCulloch-Pitts neuron is a powerful computational device. 
McCulloch and Pitts proved that a synchronous assembly of such neurons is capable in 
principle to perform any computation that an ordinary digital computer can, though not 
necessarily so rapidly or conveniently [Hertz et al 91]. 

Example 1.1: When the threshold function is used as the neuron output function, and 
binary input values 0 and 1 are assumed, the basic Boolean functions AND, OR and NOT 
of two variables can be implemented by choosing appropriate weights and threshold 
values, as shown in Figure 1.8. The first two neurons in the figure receives two binary 
inputs uj, u 2 and produces y(u/, U 2 ) for the Boolean functions AND and OR respectively. 
The last neuron implements the NOT function. 
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Figure 1.8. Implementation of Boolean functions by artificial neuron 


1.3. Network of Neurons 

While a single artificial neuron is not able to implement sotne boolean functions, the 
problem is overcome by connecting the outputs of some neurons as input to the others, so 
constituting a neural network. Suppose that we have connected tnany artificial neurons 
that we introduced in Section 1.2 to fonn a network. In such a case, there are several 
neurons in the system, so we assign indices to the neurons to discriminate between thern. 
Then to express the activation i' b neuron, the formulas are modified as follows: 

N 

a i=CEj W ji x j^ +e i O- 3 -!) 

7=1 

where xj may be either the output of a neuron determined as 

x j =fj(a j )• (1-3.2) 

or an external input detennined as: 

Xj=Uj (1.3.3) 
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In some applications the threshold value Q\ is detennined by the external inputs. Due to 
the equation (1.3.1) sometimes it may be convenient to think all the inputs are connected 
to the network only through the threshold of some special neurons called the input 
neurons. They are just conveying the input value connected to their threshold as 0 y= uj to 
their output xj with a linear output transfer function fj(a)-a. 

For a neural network we can deline a State vector x in which the i' h component is the 
output of i lh neuron, that is xp Furthermore we deline a weight matrix W, in which the 
component wjj is the weight of the connection from neuron j to neuron i. Therefore we 
can represent the system as: 

x = f (W T x +0) (1.3.4) 

Here 0 is the vector whose i th component is di and f is used to denote the vector function 
such that the function f is applied at the ft 1 component of the vector. 

Example 1.2: A simple example often given to illustrate the behavior of a neural 
networks is the one used to implement the XOR (exclusive OR) function. Notice that it 
is not possible to obtain exclusive-or or equivalence function, by using a single neuron. 
However this function can be obtained when outputs of some neurons are connected as 
inputs to some other neurons. Such a function can be obtained in several ways, only two 
of them being shown in Figure 1.9. 



Figure 1.9. Two different implementations of the exclusive-or function by using artificial neurons 
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The second neural network of Figure 1.9 can be represented as: 
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where/i and /2 being linear identity function, /3 and /4 being threshold fimctions. In the 
case of binary input, uf e{ 0 ,l} or bipolar input, that is uj e {- 1 , 1 }, all of f/ may be 
chosen as threshold function. The diagonal entries of the weight matrix are zero, since the 
neurons do not have self-feedback in this example. The weight matrix is upper triangular, 
since the network is feedforward. 

1.4. Network Architectures 


Neural computing is an alternative to programmed computing which is a mathematical 
model inspired by biological models. This computing system is made up of a number of 
artificial neurons and a huge number of interconnections between them. According to the 
structure of the connections, we identify different classes of network architectures 
(Figure 1.10). 



input hidden output input hidden output 

layer layer layer neurons neurons neurons 


Figure 1.10 a) layered feedforward neural network 
b) nonlayered recurrent neural network 
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In feedforward neural networks, the neurons are organized in the form of layers. The 
neurons in a layer get input from the previous layer and feed their output to the next 
layer. In this kind of networks connections to the neurons in the same or previous layers 
are not pennitted. The last layer of neurons is called the output layer and the layers 
between the input and output layers are called the hidden layers. The input layer is made 
up of special input neurons, transmitting only the applied extemal input to their outputs. 
In a network if there is only the layer of input nodes and a single layer of neurons 
constituting the output layer then they are called single layer network. If there are one or 
more hidden layers, such networks are called multilayer networks. 

The structures, in which connections to the neurons of the same layer or to the previous 
layers are allowed, are called recurrent networks. For a feed-forward network always 
exists an assignment of indices to neurons resulting in a triangular weight matrix. 
Furthermore if the diagonal entries are zero this indicates that there is no self-feedback on 
the neurons. However in recurrent networks, due to feedback, it is not possible to obtain a 
triangular weight matrix with any assignment of the indices. 
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